{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "(model-monitoring-overview)=\n",
    "# Model monitoring overview\n",
    "\n",
    "```{note}\n",
    "This is currently a beta feature.\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "**In this section**\n",
    "- [Architecture](#architecture)\n",
    "- [Model monitoring using the Iguazio platform interface](#model-monitoring-using-the-iguazio-platform-interface)\n",
    "- [Model monitoring using Grafana dashboards](#model-monitoring-using-grafana-dashboards)\n",
    "\n",
    "## Architecture\n",
    "The model monitoring process flow starts with collecting operational data. The operational data are converted to vectors, which are posted to the Model Server.\n",
    "The model server is then wrapped around a machine learning model that uses a function to calculate predictions based on the available vectors.\n",
    "Next, the model server creates a log for the input and output of the vectors, and the entries are written to the production data stream (a [v3io stream](https://nuclio.io/docs/latest/reference/triggers/v3iostream/)).\n",
    "While the model server is processing the vectors, a Nuclio operation monitors the log of the data stream and is triggered when a new log entry is detected.\n",
    "The Nuclio function examines the log entry, processes it into statistics which are then written to the statistics databases (parquet file, time series database and key value database).\n",
    "The parquet files are written as a feature set under the model monitoring project. The parquet files can be read either using `pandas.read_parquet` or `feature_set.get_offline_features`, like any other feature set.\n",
    "In parallel, a scheduled MLRun job runs reading the parquet files, performing drift analysis. The drift analysis data is stored so that the user can retrieve it in the Iguazio UI or in a Grafana dashboard.\n",
    "\n",
    "![Architecture](../_static/images/model-monitoring-data-flow.svg)\n",
    "\n",
    "### Drift analysis\n",
    "The model monitoring feature provides drift analysis monitoring.\n",
    "Model drift in machine learning is a situation where the statistical properties of the target variable (what the model is trying to predict) change over time.\n",
    "In other words, the production data has changed significantly over the course of time and no longer matches the input data used to train the model.\n",
    "So, for this new data, accuracy of the model predictions is low. Drift analysis statistics are computed once an hour.\n",
    "For more information see <a href=\"https://www.iguazio.com/glossary/concept-drift/\" target=\"_blank\">Concept Drift</a>.\n",
    "\n",
    "### Common terminology\n",
    "The following terms are used in all the model monitoring pages:\n",
    "\n",
    "* **Total Variation Distance** (TVD) &mdash; The statistical difference between the actual predictions and the model's trained predictions.\n",
    "* **Hellinger Distance** &mdash; A type of f-divergence that quantifies the similarity between the actual predictions, and the model's trained predictions.\n",
    "* **Kullback–Leibler Divergence** (KLD) &mdash; The measure of how the probability distribution of actual predictions is different from the second model's trained reference probability distribution.\n",
    "* **Model Endpoint** &mdash; A combination of a deployed Nuclio function and the models themselves. One function can run multiple endpoints; however, statistics are saved per endpoint.\n",
    "\n",
    "## Model monitoring using the Iguazio platform interface\n",
    "Iguazio's model monitoring data is available for viewing through the regular platform interface.\n",
    "The platform provides four information pages with model monitoring data.\n",
    "* [Model endpoint summary list](#model-endpoint-summary-list)\n",
    "* [Model endpoint overview](#model-endpoint-overview)\n",
    "* [Model drift analysis](#model-drift-analysis)\n",
    "* [Model features analysis](#model-features-analysis)\n",
    "\n",
    "1. Select a project from the project tiles screen.\n",
    "2. From the project dashboard, press the **Models** tile to view the models currently deployed.\n",
    "2. Press **Model Endpoints** from the menu to display a list of monitored endpoints.<br>\n",
    "   If the Model Monitoring feature is not enabled, the endpoints list is empty.\n",
    "\n",
    "### Model endpoint summary list\n",
    "The Model Endpoints summary list provides a quick view of the model monitoring data.\n",
    "\n",
    "![Model Monitoring Summary List](../_static/images/model_endpoints_main.png)\n",
    "\n",
    "The summary page contains the following fields:\n",
    "* **Name** &mdash; the name of the model endpoint\n",
    "* **Version** &mdash; user configured version taken from model deployment\n",
    "* **Class** &mdash; the implementation class that is used by the endpoint\n",
    "* **Model** &mdash; user defined name for the model\n",
    "* **Labels** &mdash; user configurable tags that are searchable\n",
    "* **Uptime** &mdash; first request for production data\n",
    "* **Last Prediction** &mdash; most recent request for production data\n",
    "* **Error Count** &mdash; includes prediction process errors such as operational issues (For example, a function in a failed state), as well as data processing errors\n",
    "(For example, invalid timestamps, request ids, type mismatches etc.)\n",
    "* **Drift** &mdash; indication of drift status (no drift (green), possible drift (yellow), drift detected (red))\n",
    "* **Accuracy** &mdash; a numeric value representing the accuracy of model predictions (N/A)\n",
    "\n",
    "```{note}\n",
    "Model Accuracy is currently under development.\n",
    "```\n",
    "\n",
    "### Model endpoint overview\n",
    "The Model Endpoints overview pane displays general information about the selected model.\n",
    "\n",
    "![Model Endpoints Overview](../_static/images/IG_model_endpoints_overview.png)\n",
    "\n",
    "The Overview page contains the following fields:\n",
    "* **UUID** &mdash; the ID of the deployed model\n",
    "* **Model Class** &mdash; the implementation class that is used by the endpoint\n",
    "* **Model Artifact** &mdash; reference to the model's file location\n",
    "* **Function URI** &mdash; the MLRun function to access the model\n",
    "* **Last Prediction** &mdash; most recent request for production data\n",
    "* **Error Count** &mdash; includes prediction process errors such as operational issues (For example, a function in a failed state), as well as data processing errors\n",
    "(For example, invalid timestamps, request ids, type mismatches etc.)\n",
    "* **Accuracy** &mdash; a numeric value representing the accuracy of model predictions (N/A)\n",
    "* **Stream path** &mdash; the input and output stream of the selected model\n",
    "\n",
    "Use the ellipsis to view the YAML resource file for details about the monitored resource.\n",
    "\n",
    "### Model drift analysis\n",
    "The Drift Analysis pane provides performance statistics for the currently selected model.\n",
    "\n",
    "![Model Endpoints Drift Analysis](../_static/images/IG_model_endpoints_drift_analysis.png)\n",
    "\n",
    "Each of the following fields has both sum and mean numbers displayed. For definitions of the terms see [Common Terminology](#common-terminology).\n",
    "* **TVD**\n",
    "* **Hellinger**\n",
    "* **KLD**\n",
    "\n",
    "Use the ellipsis to view the YAML resource file for details about the monitored resource.\n",
    "\n",
    "### Model features analysis\n",
    "The Features Analysis pane provides details of the drift analysis in a table format with each feature in the selected model on its own line.\n",
    "\n",
    "![Model Endpoints Features Analysis](../_static/images/IG_model_endpoints_features_analysis.png)\n",
    "\n",
    "Each field has a pair of columns. The **Expected** column displays the results from the model training phase, and the **Actual** column\n",
    "displays the results from the live production data. The following fields are available:\n",
    "* **Mean**\n",
    "* **STD** (Standard deviation)\n",
    "* **Min**\n",
    "* **Max**\n",
    "* **TVD**\n",
    "* **Hellinger**\n",
    "* **KLD**\n",
    "* **Histograms**&mdash;the approximate representation of the distribution of the data. Hover over the bars in the graph for  the details.\n",
    "\n",
    "Use the ellipsis to view the YAML resource file for details about the monitored resource.\n",
    "\n",
    "## Model monitoring using Grafana dashboards\n",
    "You can deploy a Grafana service in your Iguazio instance and use Grafana Dashboards to view model monitoring details.\n",
    "There are three dashboards available:\n",
    "* [Overview Dashboard](#model-endpoints-overview-dashboard)\n",
    "* [Details Dashboard](#model-endpoint-details-dashboard)\n",
    "* [Performance Dashboard](#model-endpoint-performance-dashboard)\n",
    "\n",
    "### Model endpoints overview dashboard\n",
    "The Overview dashboard displays the model endpoint IDs of a specific project. Only deployed models with Model Monitoring enabled are displayed.\n",
    "Endpoint IDs are URIs used to provide access to performance data and drift detection statistics of a deployed model.\n",
    "\n",
    "![overview](../_static/images/overview.png)\n",
    "\n",
    "The Overview pane provides details about the performance of all the deployed and monitored models within a project. You can change projects by choosing a new project from the\n",
    "**Project** dropdown. The Overview dashboard displays the number of endpoints in the project, the average predictions per second (using a 5-minute rolling average),\n",
    "the average latency (using a 1-hour rolling average), and the total error count in the project.\n",
    "\n",
    "Additional details include:\n",
    "* **Endpoint ID** &mdash; the ID of the deployed model. Use this link to drill down to the model performance and details panes.\n",
    "* **Function** &mdash; the MLRun function to access the model\n",
    "* **Model** &mdash; user defined name for the model\n",
    "* **Model Class** &mdash; the implementation class that is used by the endpoint\n",
    "* **First Request** &mdash; first request for production data\n",
    "* **Last Request** &mdash; most recent request for production data\n",
    "* **Error Count** &mdash; includes prediction process errors such as operational issues (for example, a function in a failed state), as well as data processing errors\n",
    "(for example, invalid timestamps, request ids, type mismatches etc.)\n",
    "* **Accuracy** &mdash; a numeric value representing the accuracy of model predictions (N/A)\n",
    "* **Drift Status** &mdash; no drift (green), possible drift (yellow), drift detected (red)\n",
    "\n",
    "At the bottom of the dashboard are heat maps for the Predictions per second, Average Latency and Errors. The heat maps display data based on 15 minute intervals.\n",
    "See [How to Read a Heat Map](#how-to-read-a-heat-map) for more details.\n",
    "\n",
    "Click an endpoint ID to drill down the performance details of that model.\n",
    "\n",
    "#### How to read a heat map\n",
    "Heat maps are used to analyze trends and to instantly transform and enhance data through visualizations. This helps to quickly identify areas of interest,\n",
    "and empower users to explore the data in order to pinpoint where there may be potential issues. A heat map uses a matrix layout with colour and shading to show the relationship between\n",
    "two categories of values (x and y axes), so the darker the cell, the higher the value. The values presented along each axis correspond to a cell which is color-coded to represent the relationship between\n",
    "the two categories. The Predictions per second heatmap shows the relationship between time, and the predictions per second, and the Average Latency per hour shows the relationship between\n",
    "time and the latency.\n",
    "\n",
    "To properly read the heap maps, follow the hierarchy of shades from the darkest (the highest values) to the lightest shades (the lowest values).\n",
    "\n",
    "```{note}\n",
    "The exact quantitative values represented by the colors may be difficult to determine. Use the [Performance Dashboard](#model-endpoint-performance-dashboard) to see detailed results.\n",
    "```\n",
    "\n",
    "### Model endpoint details dashboard\n",
    "The model endpoint details dashboard displays the real time performance data of the selected model in detail.\n",
    "Model performance data provided is rich and is used to fine tune or diagnose potential performance issues that may affect business goals.\n",
    "The data in this dashboard changes based on the selection of the project and model.\n",
    "\n",
    "This dashboard has three panes:\n",
    "\n",
    "1. [Project and model summary](#project-and-model-summary)\n",
    "2. [Analysis panes](#analysis-panes)\n",
    "   1. Overall drift analysis\n",
    "   2. Features analysis\n",
    "3. [Incoming features graph](#incoming-features-graph)\n",
    "\n",
    "![details](../_static/images/details.png)\n",
    "\n",
    "#### Project and model summary\n",
    "Use the dropdown to change the project and model. The dashboard presents the following information about the project:\n",
    "* **Endpoint ID** &mdash; the ID of the deployed model\n",
    "* **Model** &mdash; user defined name for the model\n",
    "* **Function URI** &mdash; the MLRun function to access the model\n",
    "* **Model Class** &mdash; the implementation class that is used by the endpoint\n",
    "* **Prediction/s** &mdash; the average number of predictions per second over a rolling 5-minute period\n",
    "* **Average Latency** &mdash; the average latency over a rolling 1-hour period\n",
    "* **First Request** &mdash; first request for production data\n",
    "* **Last Request** &mdash; most recent request for production data\n",
    "\n",
    "\n",
    "Use the [Performance](#model-endpoint-performance-dashboard) and [Overview](#model-endpoints-overview-dashboard) buttons view those dashboards.\n",
    "\n",
    "#### Analysis panes\n",
    "This pane has two sections: Overall Drift Analysis and Features Analysis.\n",
    "The Overall Drift Analysis pane provides performance statistics for the currently selected model.\n",
    "* **TVD** (sum and mean)\n",
    "* **Hellinger** (sum and mean)\n",
    "* **KLD** (sum and mean)\n",
    "\n",
    "\n",
    "The Features Analysis pane provides details of the drift analysis for each feature in the selected model.\n",
    "This pane includes five types of statistics:\n",
    "* **Actual** (min, mean and max) &mdash; results based on actual live data stream\n",
    "* **Expected** (min, mean and max) &mdash; results based on training data\n",
    "* **TVD**\n",
    "* **Hellinger**\n",
    "* **KLD**\n",
    "\n",
    "#### Incoming features graph\n",
    "This graph displays the performance of the features that are in the selected model based on sampled data points from actual feature production data.\n",
    "The graph displays the values of the features in the model over time.\n",
    "\n",
    "### Model endpoint performance dashboard\n",
    "Model endpoint performance displays performance details in graphical format.\n",
    "\n",
    "![performance](../_static/images/performance.png)\n",
    "\n",
    "This dashboard has five graphs:\n",
    "* **Drift Measures** &mdash; the overall drift over time for each of the endpoints in the selected model\n",
    "* **Average Latency** &mdash; the average latency of the model in 5 minute intervals, for 5 minutes and 1 hour rolling windows\n",
    "* **Predictions/s**  &mdash; the model predictions per second displayed in 5 second intervals for 5 minutes (rolling)\n",
    "* **Predictions Count** &mdash; the number of predictions the model makes for 5 minutes and 1 hour rolling windows\n",
    "\n",
    "### Configuring Grafana datasources\n",
    "Verify that you have a Grafana service running in your Iguazio MLOps Platform.\n",
    "If you do not have a Grafana service running, please follow <a href=\"https://www.iguazio.com/docs/latest-release/services/fundamentals/#create-new-service\" target=\"_blank\">Creating a Service</a> to add it to your platform.\n",
    " When you create the service: In the **Custom Parameters** tab, **Platform data-access user** parameter, select a user with access to the `/user/pipelines` directory.\n",
    "\n",
    "In addition, you will have to add access keys to your model-monitoring data source:\n",
    "   1. Open your Grafana service.\n",
    "   2. Navigate to **Configuration | Data Sources**.\n",
    "   3. Press **model-monitoring**.\n",
    "   4. In Custom HTTP Headers, configure the cookie parameter. Set the value of `cookie` to:\n",
    "   `session=j:{\"sid\": \"<YOUR ACCESS KEY>\"}`\n",
    "   5. Press **Save & Test** for verification. You'll receive a confirmation with either a success or a failure message.\n",
    "\n",
    "<img src=\"../_static/images/model-monitoring-datasource.png\" alt=\"Grafana Model Monitoring Datasource\" width=\"400\"/><br>\n",
    "\n",
    "\n",
    "### Configuring Grafana dashboards\n",
    "From Iguazio 3.5.3, the overview, details, and performance dashboards can be found under **Dashboards | Manage | private**.\n",
    "You can also import the latest dashboards versions by downloading them using the following links:\n",
    "\n",
    "**Iguazio 3.5.3**\n",
    " * {download}`Model Monitoring - Overview <./dashboards/model-monitoring-overview.json>`\n",
    " * {download}`Model Monitoring - Details <./dashboards/model-monitoring-details.json>`\n",
    " * {download}`Model Monitoring - Performance <./dashboards/model-monitoring-performance.json>`\n",
    "\n",
    "**Iguazio 3.5.2 and older**\n",
    " * {download}`Model Monitoring - Overview <./dashboards/iguazio-3.5.2-and-older/model-monitoring-overview.json>`\n",
    " * {download}`Model Monitoring - Details <./dashboards/iguazio-3.5.2-and-older/model-monitoring-overview.json>`\n",
    " * {download}`Model Monitoring - Performance <./dashboards/iguazio-3.5.2-and-older/model-monitoring-overview.json>`\n",
    "\n",
    "Upload dashboards to your Grafana service by:\n",
    "   1. Navigate to your Grafana service in the Services list and press it.\n",
    "   2. Press the dashboards icon in left menu.\n",
    "   3. In the Dashboard Management screen, press **IMPORT**, and select one file to import. Repeat this step for each dashboard.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "\n",
    "\n",
    "```{note}\n",
    "You need to train and deploy a model to see results in the dashboards.\n",
    "The dashboards immediately display data if you already have a model that is trained and running with production data.\n",
    "```"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3.8.10 64-bit",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.10"
  },
  "vscode": {
   "interpreter": {
    "hash": "916dbcbb3f70747c44a77c7bcd40155683ae19c65e1c03b4aa3499c5328201f1"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
